Estimation of a k−monotone density, part 3: limiting Gaussian versions of the problem; invelopes and envelopes

Let k be a positive integer. The limiting distribution of the nonparametric maximum likelihood estimator of a k−monotone density is given in terms of a smooth stochastic process Hk described as follows: (i) Hk is everywhere above (or below) Yk, the k− 1 fold integral of two-sided standard Brownian motion plus (k!/(2k)!)t2k when k is even (or odd). (ii) H(2k−2) k is convex. (iii) Hk touches Yk at exactly those points where H(2k−2) k has changes of slope. We show that Hk exists if a certain conjecture concerning a particular Hermite interpolation problem holds. The process H1 is the familiar greatest convex minorant of two-sided Brownian motion plus (1/2)t2, which arises in connection with nonparametric estimation of a monotone (1-monotone) function. The process H2 is the “invelope process” studied in connection with nonparametric estimation of convex functions (up to a scaling of the drift term) in Groeneboom, Jongbloed, and Wellner (2001a). We therefore refer to Hk as an “invelope process” when k is even, and as an “envelope process” when k is odd. We establish existence of Hk for all non-negative integers k under the assumption that our key conjecture holds, and study basic properties of Hk and its derivatives. Approximate computation of Hk is possible on finite intervals via the iterative (2k−1) spline algorithm which we use here to illustrate the theoretical results. 1 Research supported in part by National Science Foundation grant DMS-0203320 2 Research supported in part by National Science Foundation grants DMS-0203320, and NIAID grant 2R01 AI291968-04 AMS 2000 subject classifications. Primary: 62G05; secondary 60G15, 62E20.